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DETAILED ACTION 



Claim Rejections - 35 USC § 101 

1. The text of those sections of Title 35, U.S. Code not included in this action can 

be found in a prior Office action. 

The USPTO "Interim Guidelines for Examination of Patent Applications for Patent 
Subject Matter Eligibility" (Official Gazette notice of 22 November 2005), Annex IV, 
reads as follows: 

Descriptive material can be characterized as either "functional descriptive material" or "nonfunctional 
descriptive material." In this context, "functional descriptive material" consists of data structures and 
computer programs which impart functionality when employed as a computer component. (The 
definition of "data structure" is "a physical or logical relationship among data elements, designed to 
support specific data manipulation functions." The New IEEE Standard Dictionary of Electrical and 
Electronics Terms 308 (5th ed. 1993).) "Nonfunctional descriptive material" includes but is not limited 
to music, literary works and a compilation or mere arrangement of data. 

When functional descriptive material is recorded on some computer-readable medium it becomes 
structurally and functionally interrelated to the medium and will be statutory in most cases since use 
of technology permits the function of the descriptive material to be realized. Compare In re Lowry, 32 
F.3d 1579, 1583-84, 32 USPQ2d 1031, 1035 (Fed. Cir. 1994) (claim to data structure stored on a 
computer readable medium that increases computer efficiency held statutory) and Warmerdam, 33 
F.3d at 1360-61, 31 USPQ2d at 1759 (claim to computer having a specific data structure stored in 
memory held statutory product-by-process claim) with Warmerdam, 33 F.3d at 1361, 31 USPQ2d at 
1760 (claim to a data structure per se held nonstatutory). 

In contrast, a claimed computer-readable medium encoded with a computer program is a computer 
element which defines structural and functional interrelationships between the computer program and 
the rest of the computer which permit the computer program's functionality to be realized, and is thus 
statutory. See Lowry, 32 F.3d at 1583-84, 32 USPQ2d at 1035. 



Claim 22 is rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter as follows. Claim 22 defines a "system". 
However, while the preamble defines a "system", which would typically be indicative of 
an "apparatus", the body of the claim lacks definite structure indicative of a physical 
apparatus. Furthermore, the specification indicates that the invention may be 
embodied as pure software page 19, paragraph [0084]. Therefore, the claim as a 
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whole appears to be nothing more than a "system" of software elements, thus defining 
functional descriptive material per se. 

Functional descriptive material may be statutory if it resides on a "computer- 
readable medium or computer-readable memory". The claim(s) indicated above lack 
structure, and do not define a computer readable medium and are thus non-statutory 
for that reason (i.e., "When functional descriptive material is recorded on some 
computer-readable medium it becomes structurally and functionally interrelated to the 
medium and will be statutory in most cases since use of technology permits the 
function of the descriptive material to be realized" - Guidelines Annex IV). The scope 
of the presently claimed invention encompasses products that are not necessarily 
computer readable, and thus NOT able to impart any functionality of the recited 
program. The examiner suggests: 

1. Amending the claim(s) to embody the program on "computer-readable 
medium" or equivalent; assuming the specification does NOT define the computer 
readable medium as a "signal", "carrier wave", or "transmission medium" which are 
deemed non-statutory; or 

2. Adding structure to the body of the claim that would clearly define a 
statutory apparatus. 

Any amendment to the claim should be commensurate with its corresponding 
disclosure. 



Application/Control Number: 10/757,490 Page 4 

Art Unit: 2624 

Note: 

"A transitory, propagating signal ... is not a "process, machine, manufacture, or 
composition of matter." Those four categories define the explicit scope and reach of 
subject matter patentable under 35 U.S.C. § 101; thus, such a signal cannot be 
patentable subject matter." {In re PetrusA.C.M. Nuijten; Fed Cir, 2006-1371, 
9/20/2007). 

Should the full scope of the claim as properly read in light of the disclosure 
encompass non-statutory subject matter such as a "signal", the claim as a whole would 
be non-statutory. Should the applicant's specification define or exemplify the computer 
readable medium or memory (or whatever language applicant chooses to recite a 
computer readable medium equivalent) as statutory tangible products such as a hard 
drive, ROM, RAM, etc, as well as a non-statutory entity such as a "signal", "carrier 
wave", or "transmission medium", the examiner suggests amending the claim to include 
the disclosed tangible computer readable storage media, while at the same time 
excluding the intangible transitory media such as signals, carrier waves, etc. 

Cla im Rejections - 35 USC § 103 

2. The text of those sections of Title 35, U.S. Code not included in this action can 
be found in a prior Office action. 

3. Claims 1, 5-7, 21 and 22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Guo et al. (Hereinafter "Guo": Ping Guo and Michael R. Lyu, "A Study 
on Color Space Selection for Determining Image Segmentation and Region Number", 
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Proceedings of the international Conference, IC-AI 2000, CSREA Press, Athens, GA, 
USA, Vol.3, pp. 1127-1132, June 2000) in view of Reavy et al. (US 6,788,308 B2). 

Regarding claims 1 and 22, Guo teaches a method and analogous system for 
automatically determining a foreground color for a digital image, comprising: 

(a) automatically dividing the colors of the pixels of at least a part of the digital 
image into a number of color clusters in a color space (see Abstract); 

(b) automatically segmenting the part of the digital image into regions according 
to the color clusters (§ 1, U 1 and § 3, H 3). 

Guo does not teach the limitation of (c) automatically grouping the color clusters 
into interference clusters comprising a pixel in a selected region of the image and 
benign clusters comprising no pixel in the selected region. 

Reavy discloses a method for determining a foreground color for an image 
comprising: 

(c) automatically grouping the color clusters into interference clusters 
comprising a pixel in a selected region of the image and benign clusters comprising no 
pixel in the selected region (column 2, lines 21-23, i.e. area covered by text 
versus area not covered by text). 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to use Reavy's grouping method in Guo's system in order to improve 
readability of text (Reavy: column 1, lines 39-40). 
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Guo combined with Reavy teaches the method above comprising: 

(d) automatically selecting a color being related to all interference clusters 
according to predetermined criteria (Reavy: column 2, lines 7-14); and 

(e) one of automatically displaying, storing and communicating data representing 
the selected foreground color (Reavy: Fig. 4, Display Monitor 26). 

Regarding claim 5, Guo combined with Reavy teaches the method according 
to claim 1, wherein said dividing at (a) comprises converting the image data to a 
predetermined color format (Guo: § 2.4, H 2). 

Regarding claim 6, Guo combined with Reavy teaches the method according 
to claim 1, wherein said dividing at (a) comprises using an Expectation-Maximization 
clustering (Guo: § 2.1). 

Regarding claim 7, Guo combined with Reavy teaches the method according 
to claim 1, wherein said dividing at (a) comprises determining the number of clusters 
using a model selection method one of a Bayesian Information Criterion and a Universal 
Model-based Minimum Description Length Principle (Guo: § 2.3, i.e. Bayesian 
Probabilistic Classification). 

Regarding claim 21, Guo combined with Reavy teaches the method according 
to claim 1, further comprising one of displaying and storing a predetermined object 
using the selected foreground color together with the digital image (Guo: § 3 and Fig. 
1). 
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4. Claim 2 is rejected under 35 U.S.C. 103(a) as being unpatentable over Guo 
combined with Reavy as applied to claim 1 above, in view of Hamlin et al. (Hereinafter 
"Hamlin": US 5473738), and further in view of Sandow (US 5909220). 

Guo combined with Reavy teaches the method according to claim 1. Guo 
combined with Reavy does not teach the limitation wherein said selecting at (b) further 
comprises: selecting a harmonious color set with respect to the color clusters. 

However, Hamlin discloses a method wherein said selecting at (b) further 
comprises: selecting a harmonious color set with respect to the color clusters (Column 
1, lines 9-12). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Hamlin's harmonious color selecting method with the 
Guo/Reavy foreground color determining method, the motivation being to produce a 
pleasing effect to the eye (Hamlin: column 2, lines 25-27). 

Guo combined with Reavy and Hamlin does not teach the limitation wherein said 
selecting at (b) further comprises testing the harmonious color set for legibility. 

However Sandow discloses a method wherein said selecting at (b) further 
comprises testing the harmonious color set for legibility (column 17, lines 25-29). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Sandow's legibility testing method with the above 
combined teaching of Guo, Reavy and Hamlin, the motivation being to control the 
legibility of lettering (Sandow: Column 3, lines 30-34). 
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5. Claims 3 and 4 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Guo combined with Reavy, Hamlin and Sandow as applied to claim 2 above, and further 
in view of You et al. (Hereinafter "You": J. You, E. Pissaloux, H. A. Cohen, "A 
Hierarchical Image Matching Scheme Based on the Dynamic Detection of Interesting 
Points", International Conference on Speech and Image Processing, 1995, Vol. 4, pp. 
2467-2470). 

Regarding claim 3, Guo in combination with Reavy, Hamlin and Sandow 
teaches the method according to claim 2, as disclosed above, wherein said testing the 
harmonious color set for legibility further comprises: computing local measures of 
contrast between background and foreground in a neighborhood of a predetermined 
foreground region (Sandow: column 19, lines 1-8 and Fig. 12, background 65 in 
a neighborhood of foreground 64). 

Guo in combination with Reavy, Hamlin and Sandow does not teach the 
limitation wherein said testing the harmonious color set for legibility further comprises 
computing a legibility score representative of a lowest few contrast values observed 
over the predetermined foreground region. 

However, You discloses computing a legibility score representative of a lowest 
few contrast values observed over the predetermined foreground region (page 2468, 
§ 2.1 and § 3, H 2, i.e. computing the Hausdorff distance (legibility score) 
using only the "interesting points" over the foreground region). 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine You's computing method with the above combined 
teaching of Guo, Reavy, Hamlin and Sandow, the motivation being to decrease 
computation time (You: § 3, U 1). 

Regarding claim 4, Guo in combination with Reavy, Hamlin and Sandow 
teaches the method according to claim 2, wherein the foreground color is selected 
according to a legibility criterion for a predetermined foreground region by: computing 
local measures of contrast between background and foreground in a neighborhood for a 
predetermined foreground region (Sandow: column 19, lines 1-8 and Fig. 12, 
background 65 in a neighborhood of foreground 64). 

Guo in combination with Reavy, Hamlin and Sandow does not teach the 
limitation of computing a legibility score representative of a lowest few contrast values 
observed over the predetermined foreground region. 

However You discloses computing a legibility score representative of a lowest 
few contrast values observed over the predetermined foreground region (page 2468, 
§ 2.1 and § 3, U 2, i.e. computing the Hausdorff distance (legibility score) 
using only the "interesting points" over the foreground region). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine You's computing method with the above combined 
teaching of Guo, Reavy, Hamlin and Sandow, the motivation being to decrease 
computation time (You: § 3, U 1). 
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6. Claim 9 is rejected under 35 U.S.C. 103(a) as being unpatentable over Guo 
combined with Reavy as applied to claim 1 above, in view of Shi et al. (Hereinafter 
"Shi": Jianbo Shi and Jitendra Malik, "Normalized Cuts and Image Segmentation", 2000, 
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 8, pp. 
888-905). 

Guo combined with Reavy teaches the method according to claim 1. 

Guo combined with Reavy does not teach the limitation wherein the segmenting 
comprises using one of a normalized cut criterion and an energy-minimization method. 

However, Shi discloses a segmentation method using one of a normalized cut 
criterion and an energy-minimization method (See Abstract). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Shi's criterion with the Guo/Reavy teaching, the 
motivation being to measure the goodness of an image segment (Shi: page 889, left 
column, lines 6-8). 

7. Claim 11 is rejected under 35 U.S.C. 103(a) as being unpatentable over Guo 
combined with Reavy as applied to claim 1 above, in view of Sandow. 

Guo combined with Reavy teaches the method according to claim 1. 

Guo combined with Reavy does not teach the limitation wherein the foreground 
color is selected according to a legibility criterion for a predetermined foreground 
region. 
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However, Sandow discloses a method wherein the foreground color is selected 
according to a legibility criterion for a predetermined foreground region (column 6, 
lines 55-60). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Sandow's legibility testing method with the above 
teaching of Guo, the motivation being to control the legibility of lettering (Sandow: 
Column 3, lines 30-34). 

8. Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Guo 
combined with Reavy and Sandow as applied to claim 11 above, in view of Gatica-Perez 
(Hereinafter "Gatica-Perez": Daniel Gatica-Perez, Zhi Zhou, Ming-Ting Sun and Vincent 
Hsu, "Video Object Hyper-Links for Streaming Applications", 2002, Springer- Verlag 
Berlin Heidelberg, VISUAL 2002, LNCS 2314, pp, 229-238). 

Guo in combination with Reavy and Sandow teaches the method according to 
claim 11. 

Guo in combination with Reavy and Sandow does not teach the limitation 
wherein the foreground color is selected based on a likelihood ratio of the hypothesis 
that the digital image contains the foreground region and the hypothesis that the digital 
image does not contain the foreground region. 

However, Gatica-Perez discloses a method wherein the foreground color is 
selected based on a likelihood ratio of the hypothesis that the digital image contains the 
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foreground region and the hypothesis that the digital image does not contain the 
foreground region (page 232 and Fig. 2). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Gatica-Perez's color selection method with the above 
combined teaching of Guo, Reavy and Sandow, the motivation being to reduce the 
search space for segmentation (Gatica-Perez: § 2.3, H 1, lines 1-3). 
9. Claim 13 is rejected under 35 U.S.C. 103(a) as being unpatentable over Guo 
combined with Sandow, in view of Gatica-Perez as applied to claim 12 above, and 
further in view of You. 

Guo in combination with Reavy and Sandow and further in view of Gatica-Perez 
teaches the method according claim 12. 

Guo in combination with Reavy and Sandow and further in view of Gatica-Perez 
does not teach the limitation wherein selecting the foreground color comprises 
computing the legibility of 11 min x T max y C 2 r(x+y), wherein 12 r ( x ) = h log Pr ( I 
( x ) | T ) Pr ( I ( x ) | B ) ,C.sub..epsilon..sup.2 is a disc of radius .epsilon and wherein 
Pr(I(x).vertline.T) denotes heuristic or other models of likelihoods that the image I 
contains text T at a given pixel x and Pr(I(x).vertline.B) denotes heuristic or other 
models of likelihoods that the image I contains background B at the given pixel x. 

However, You discloses a method wherein selecting the foreground color 
comprises computing the legibility of 11 min x T max y C 2 r(x+y), wherein 12 r ( x ) = 
h log Pr ( I ( x ) | T ) Pr ( I ( x ) | B ) ,C.sub.. epsilon. .sup. 2 is a disc of radius .epsilon 
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and wherein Pr(I(x).vertline.T) denotes heuristic or other models of likelihoods that the 
image I contains text T at a given pixel x and Pr(I(x).vertline.B) denotes heuristic or 
other models of likelihoods that the image I contains background B at the given pixel x 
(page 2468, § 2.1 and § 3, U 2, i.e. computing the Hausdorff distance 
(legibility score) using only the "interesting points" over the foreground 
region, and using Gatica-Perez's likelihood ratio disclosure (Gatica-Perez: 
page 232 and Fig. 2)). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine You's computing method with the above combined 
teaching of Guo, Reavy, Sandow and Gatica-Perez, the motivation being to decrease 
computation time (You: § 3, H 1). 

10. Claims 14-16 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Guo combined with Reavy as applied to claim 1 above, in view of Hamlin. 

Regarding claim 14, Guo combined with Reavy teaches the method according 
to claim 1. Guo combined with Reavy does not teach the limitation wherein the 
foreground color is selected according to a color harmony criterion. 

However, Hamlin discloses a method wherein the foreground color is selected 
according to a color harmony criterion (column 1, lines 9-12). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Hamlin's harmonious color selecting method with the 



Application/Control Number: 10/757,490 Page 14 

Art Unit: 2624 

Guo/Reavy foreground color determining method, the motivation being to produce a 
pleasing effect to the eye (Hamlin: column 2, lines 25-27). 

Regarding claim 15, Guo in combination with Reavy and Hamlin teaches the 
method according to claim 14, wherein the foreground color is selected according to at 
least one of a monotonic, a complementary, and a p-adic color harmony criterion in HSL 
space (Hamlin: column 4, lines 25-27). 

Regarding claim 16, Guo combined with Reavy teaches the method according 
to claim 14, wherein the foreground color is selected according to a color harmony 
criterion with respect to at least one interference cluster (Reavy: column 2 lines 9- 
14, i.e. background color as determined by Guo's clustering method). 
11. Claims 19 and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Guo combined with Reavy as applied to claim 1 above, in view of Hamlin, and 
further in view of Sandow. 

Regarding claim 19, Guo combined with Reavy teaches the method according 
to claim 1. Guo combined with Reavy does not teach the limitation wherein said 
selecting at (d) comprises determining a color subset according to a color harmony 
criterion and maximizing a legibility function in the color subset. 

However, Hamlin discloses a method wherein said selecting at (d) further 
comprises selecting a harmonious subset according to a color harmony criterion 
(column 7, lines 19-22). 
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Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Hamlin's harmonious color selecting method with the 
Guo/Reavy foreground color determining method, the motivation being to produce a 
pleasing effect to the eye (Hamlin: column 2, lines 25-27). 

Guo combined with Reavy and Hamlin does not teach the limitation wherein said 
selecting at (d) further comprises maximizing a legibility function in the color subset. 

However Sandow discloses a method wherein said selecting at (d) further 
comprises maximizing a legibility function in the color subset (column 17, lines 25-29 
and column 36, lines 36-49, i.e. selecting color subset for maximum legibility 
slider value). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to combine Sandow's legibility maximization method with the 
above combined teaching of Guo, Reavy and Hamlin, the motivation being to control 
the legibility of lettering (Sandow: Column 3, lines 30-34). 

Regarding claim 20, Guo combined with Reavy, Hamlin and Sandow teaches the 
method according to claim 1 and the harmonious color selection method (i.e. 
function) and legibility maximization function as recited in claim 19 above. 

Guo combined with Reavy, Hamlin and Sandow does not disclose the limitation 
wherein the foreground color c is selected for which 13i = lMll(c,Pi) + k= lNk 
h ( c , K k ) is maximal, wherein c denotes the foreground color, P.sub.i denote the 
interference clusters, K.sub.k denote all clusters, both benign and interference, I is a 
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legibility function in color space, h is a color harmony function, and .alpha.. sub. i and 
.gamma.. sub. k are weighting factors. 

However, the examiner takes official notice that maximizing a weighted sum was 
notoriously well know in the art at the time of the invention. 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to maximize a weighted sum comprising the Guo/Reavy/Hamlin 
harmonious color selection function and Sandow's legibility maximization function, the 
motivation being to combine optimization of a pleasing effect to the eye (Hamlin: 
column 2, lines 25-27) and the legibility of lettering (Sandow: Column 3, lines 
30-34). 

Summary of Applicant's Remarks: Lovelady does not teach features related 
to "selecting a foreground color related to an interference cluster", as now recited in 
claim 1. 

Examiner's Response: Applicant's argument is moot in view of the new 
grounds of rejection applied to claim 1 above. 

Allowable Subject Matter 

12. Claims 17 and 18 are objected to as being dependent upon a rejected base 
claim, but would be allowable if rewritten in independent form including all of the 
limitations of the base claim and any intervening claims. 
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Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to CLIFTON G. DALEY whose telephone number is 571- 
270-3144. The examiner can normally be reached on Monday - Friday 7:30am - 
4:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Samir Ahmed can be reached on 571-272-7413. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. Status 
information for unpublished applications is available through Private PAIR only. For 
more information about the PAIR system, see http://pair-direct.uspto.gov. Should you 
have questions on access to the Private PAIR system, contact the Electronic Business 
Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO 
Customer Service Representative or access to the automated information system, call 
800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Samir Ahmed 
Examiner 
Art Unit 2624 

CGD 

8/4/2008 
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/Samir A. Ahmed/ 

Supervisory Patent Examiner, Art Unit 2624 



